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Abstract 

In this paper, we address the problem of estimating and 
removing non-uniform motion blur from a single blurry im¬ 
age. We propose a deep learning approach to predicting 
the probabilistic distribution of motion blur at the patch 
level using a convolutional neural network (CNN). We fur¬ 
ther extend the candidate set of motion kernels predicted 
by the CNN using carefully designed image rotations. A 
Markov random field model is then used to infer a dense 
non-uniform motion blur field enforcing motion smooth¬ 
ness. Finally, motion blur is removed by a non-uniform de- 
blurring model using patch-level image prior. Experimental 
evaluations show that our approach can effectively estimate 
and remove complex non-uniform motion blur that is not 
handled well by previous approaches. 

1. Introduction 

Image deblurring El [6j [9j |TT 1 [14] QjD [23] [30) aims at 
recovering sharp image from a blurry image due to camera 
shake, object motion or out-of-focus. In this paper, we focus 
on estimating and removing spatially varying motion blur. 

Non-uniform deblurring [ 10] [13] [201 has attracted much 
attention in recent years. Methods in mmmm work 
on non-uniform blur caused by camera rotations, in-plane 
translations or forward out-of-plane translations. They are 
effective for removing non-uniform blur consistent with 
these motion assumptions. Another category of approaches 
works on non-uniform motion blur caused by object mo¬ 
tion. They estimate blur kernels by analyzing image statis¬ 
tics 1021, blur spectrum (T), or with a learning approach us¬ 
ing hand-crafted features 0 Other approaches [13., 29] 
jointly estimate the sharp image and blur kernels using 
a sparsity prior. It is still challenging today to remove 
strongly non-uniform motion blur captured in complex 
scenes. 

In this work, we propose a novel deep learning-based ap¬ 
proach to estimating non-uniform motion blur, followed by 
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a patch statistics-based deblurring model adapted to non- 
uniform motion blur. We estimate the probabilities of mo¬ 
tion kernels at the patch level using a convolutional neural 
network (CNN) l5l [l2l 15, 16 ], then fuse the patch-based 
estimations into a dense field of motion kernels using a 
Markov random field (MRF) model. To fully utilize the 
CNN, we propose to extend the candidate motion kernel 
set predicted by CNN using an image rotation technique, 
which significantly boost its performance for motion kernel 
estimation. Taking advantage of the strong feature learning 
power of CNNs, we can well predict the challenging non- 
uniform motion blur that can hardly be well estimated by 
the state-of-the-art approaches. 

Figure [I] illustrates our approach. Given a blurry image, 
we first estimate non-uniform motion blur field by a CNN 
model, then we deconvolve the blurry image. Our approach 
can effectively estimate the spatially varying motion ker¬ 
nels, which enable us to well remove the motion blur. 

1.1. Related Work 

Estimating accurate motion blur kernels is essential to 
non-uniform image deblurring. In 018, 25] 26, 31 j, non- 
uniform motion blur is modeled as a global camera motion, 
which basically estimates an uniform kernel in the camera 
motion space. Methods in [Kj|[T3]|29’] jointly estimate the 
motion kernels and sharp image. They rely on a sparsity 
prior to infer the latent sharp image for better motion kernel 
estimation. Different to them, we estimate motion blur ker¬ 
nels directly using the local patches, which does not require 
the estimation of camera motion or a latent sharp image. 

Another category of approaches (T]0| estimates spatially 
varying motion blur based on local image features. The 
method in m estimates motion blur based on blur spectrum 
analysis of image patch in Fourier transform space. rm 
predicts motion blur kernel using natural image statistics. 
0 estimates motion blur by analyzing the alpha maps of 
image edges. 0 learns a regression function to predict mo¬ 
tion blur kernel based on some hand-crafted features. Dif¬ 
ferent to them, we estimate motion blur kernels using a con¬ 
volutional neural network, followed by a carefully designed 
motion kernel extension method and MRF model to predict 
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(b) Estimated motion blur field by CNN 



(c) Result after deblurring 


Figure 1. An example illustrating our approach. Given an image with non-uniform motion blur (left). We first estimate the field of 
non-uniform motion blur kernels by a convolutional neural network (middle), then deconvolve the blurred image (right). 


a dense field of motion kernels. Our approach can well esti¬ 
mate complex and strong motion blur, which can hardly be 
well estimated by the previous approaches. 

Recently, there has been some related work on learning- 
based deblurring approaches. ED proposes a discrimina¬ 
tive deblurring approach using cascade of Gaussian CRF 
models for uniform blur removal. l22l proposes a neural 
network approach for learning a denoiser to suppress noises 
during deconvolution. f28l designs an image deconvolu¬ 
tion neural network for non-blind deconvolution. These 
approaches above focus on designing better learning-based 
model for uniform blur removal. Our approach works on 
a more challenging task of non-uniform motion blur esti¬ 
mation and removal. Our CNN-based approach provides an 
effective method for solving this problem. 

2. Learning a CNN for Motion Blur Estimation 


Before giving the details of our approach, let us first 
introduce our general formulation for non-uniform motion 
blur. We consider non-uniform image blur caused by object 
or camera motion. Given a blurry image /, we represent 
the local motion blur kernel at an image pixel pE fi(D is 
the image region) by a motion vector m p = ( l v ,o v ), which 
characterizes the length and orientation of the motion field 
in p when the camera shutter is open. As shown in Fig.J^a), 
each motion vector determines a motion kernel with non¬ 
zero values only along the motion trace. The blurry image 
can then be represented by I = k(M ) * Iq, i.e., the con¬ 
volution of a latent sharp image Iq with the non-uniform 
motion blur kernels k(M) determined by the motion field 
M = {m p } pe n. 

In the following paragraph, we also represent the motion 
vector m p as (u p , v p ) in Cartesian coordinate system based 
on the transform: 


We propose to estimate spatially-varying motion blur 
kernels using a convolutional neural network. The basic 
idea is that we first predict the probabilities of different mo¬ 
tion kernels for each image patch. Then we estimate dense 
motion blur kernels for the whole image using a Markov 
random field model enforcing motion smoothness. 



(c) Candidate motion kernel set for learning CNN 


Figure 2. Representation of motion blur kernel by motion vector 
and generation of motion kernel candidates. 


u p = l p cos(o p ),Vp = l p sin(op). (1) 

The estimation of spatially-varying motion blur kernels 
is equivalent to estimating the motion fielcf] from a single 
blurry image. In our approach, we do not make any global 
parametric assumptions (e.g., homography) on the motion, 
therefore the motion kernel estimation is challenging, and 
we only use local image regions for predicting these kernels. 

2.1. Patch-level Motion Kernel Estimation by CNN 

We now present our approach to predicting motion blur 
kernels (or equivalently, the motion vector) at the patch 
level. We decompose the image into overlapping patches 
of size 30 x 30. Given a blurry patch ^ p centered at pixel 
p, we aim to predict the probabilistic distribution of motion 
kernels: 

P(m = (/,o)|^) (2) 

for all l e S l and o G S°, S l and S° are the sets of motion 
lengths and orientations respectively. In the followings, we 

] Note that motions m = (/, o) and m' = (Z, o + 180°) generate the 
same motion blur kernel. We therefore only need to estimate the motions 
with o G [0,180°). 
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Figure 3. Structure of CNN for motion kernels prediction. It is composed of 6 layers of convolutional layers and fully connected layers. It 
outputs the probability of each candidate motion kernel using soft-max layer. The right sub-figure shows the learned filters in Cl. 


call this distribution as motion distribution. 

Taking the problem of motion kernel estimation as a 
learning problem, we utilize convolutional neural network 
to learn the effective features for predicting motion distri¬ 
butions in Eqn. ([2]). We generate a set of candidate motion 
kernels by discretizing the motion space, i.e., the ranges of 
length and orientation of the motion vectors. In our imple¬ 
mentation, we discretize the range of motion length into 13 
samples from l = 1 to 25 with interval of two, and dis¬ 
cretize the range of motion orientation [0,180°) into 6 sam¬ 
ples from 0° to 150° with interval of 30°. Note that when 
the motion length l = 1, all motion vectors correspond to 
the same blur kernel (i.e., identity kernel) on image grid re¬ 
gardless of the motion orientation. We therefore generate 
73 candidate motion vectors (shown in Fig.[2|c)) in differ¬ 
ent combinations of motion lengths and orientations. We 
denote the above set of motion kernel candidates as S and 
the sets of motion lengths and motion orientations as S l and 
S° respectively. Obviously, these candidate motion vectors 
are far from dense in the continuous motion space. In Sec¬ 
tion [2]2] we will show how to extend the motion kernels of 
CNN to predict motion kernels outside the set 5. 

Given the candidate motion kernel set 5, we next con¬ 
struct and learn CNN for predicting the motion distribution 
over S given a blurry patch. The convolutional neural net¬ 
work is constructed as follows. As shown in Fig. [3] the net¬ 
work has six layers: Cl — M2 — C3 — M4 — F5 — 56. Cl 
is a convolutional layer using filters (7x7x3) followed by 
ReFU (i.e., f{pc) *= max(x, 0) |[T5ll ) non-linear transform; 
M2 is a max-pooling layer over 2x2 cells with stride 2; 
C3 is a convolutional layer using 256 filters (5 x 5 x 96); 
M 4 is a max-pooling layer same as M2; F 5 is a fully con¬ 
nected layer with 1024 neurons; 56 is a soft-max layer with 
73 labels, and each label corresponds to a candidate motion 
blur kernel in 5 as shown in Fig.[2jc). 

To train the CNN model, we generate a large set of train¬ 
ing data T = {4//-, rrifc}^ =1 , which are composed of blurry 
patch / motion kernel pairs. We synthetically generate 
blurry images by convolving clean natural images with the 
73 possible motion kernels, then randomly crop 30 x 30 x 3 
color patches from the blurry images as the training patches 
{^k}k=i’ an ^ take the labels of corresponding ground-truth 
motion kernels as the training labels {rrifc}^ =1 . We gener¬ 


ate training data using 1000 images randomly sampled from 
PASCAF VOC 2010 database and finally construct a train¬ 
ing set of around 1.4 million pairs of blurry patches and 
their ground-truth motion kernels. Using Caffe BT0 we 
train the CNN model in one million iterations by stochas¬ 
tic gradient descent algorithm with batches of 64 patches in 
each iteration. 

Because the final layer of the CNN is a soft-max layer, 
we can predict the probabilities of motion kernels given an 
observed blurry patch 4> as 


P(m = (i,o)|¥) 


exp ((wf 6 ) T (/> F5 (^)) 

E n exp(K6)T^ 5( ^))’ 


(3) 


where ref 6 is the vector of weights on neuron connections 
from F5 layer to the neuron in 56 layer representing the 
motion kernel (/, o), c is the index of (/, o) in 5. 4*) is 

the output features of F 5 layer of a blurry patch 4/, which 
is a 1024-dimensional feature vector. 

In our implementation, we also tried to learn more com¬ 
plex CNN structures (e.g., with one more convolutional 
layer or more filters in convolutional layers), but the learn¬ 
ing speed is significantly slower while the final prediction 
results are not significantly improved. Figure 0 (right) 
shows examples of automatically learned filters by our CNN 
model for motion kernel prediction. These filters reflect di¬ 
verse local structures in sharp or blurry patch instances. 



Figure 4. Motion kernel estimation on a rotated patch. I is a blurry 
image, Re I is the rotated image with 6 (0 = —24° in this case). 

2.2. Extending the Motion Kernel Set of CNN 


Our learned CNN model can predict the probabilities 
of 73 candidate motion kernels in 5. Obviously, they are 
not sufficiently dense in the motion space. We next extend 
the motion kernel set predicted by the CNN to enable the 
prediction for motion kernels outside 5. 

2 http: // caffe .berkeley vision. org 
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Predicting probabilities of motion kernels by CNN 
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Figure 5. Extension of motion kernel set predicted by CNN using rotated images. For an image I, we generate its rotated images 
R-qoI : -R— 12 ° 1 1 R-i8°/, R- 24 °F, then feed each patch and its rotated versions into the CNN to predict motion distributions. By con¬ 
catenating all the motion distribution estimations, we can estimate the probabilities of more densely sampled motion kernels. 


We make the extension based on the following observa¬ 
tion. As shown in Fig. |4j given a blurry image /, we rotate 
it by 0 degrees (denoted as Re I, Re is a rotation operator). 
For a pair of patches 4/ p (J) and its rotated version ^ p (RqI) 
cropped from I and Re I centered at pixel p respectively, 
if we can predict that the motion kernel of Sk p (ReI) is 
m = (l,o), then we can deduce directly that the motion 
kernel of corresponding patch & P (I) in I is m = ( l,o — 0 ). 

Based on the above observation, we can estimate the 
probabilities of motion kernels for patch ^f p (I) using its 
rotated patch ty p (ReI). By feeding the rotated patch into 
CNN, we can estimate the probabilities of motion kernels 
for the rotated patch: P(m = (l,o)\'& p (ReI)),m G S, 
then we can deduce that the motion distribution of the orig¬ 
inal patch \F P (J) before rotation is: 

P(m = (Z,o - 0)|tf p (I)) = P(m = (l,o)\%(RoI)). (4) 

Note that motion m = (/, o — 6) may not belong to the 
motion kernel set of CNN (i.e., S). 

By carefully designing the image rotations, we can 
extend the motion kernel set of CNN as follows. Re¬ 
member that the original CNN can predict probabilities 
of 73 motion kernels in S with orientations in S° = 
{0°,30°,60°,90°, 120°, 150°} with interval of 30°. As 
shown in Fig.[5j given a blurry image /, we first generate its 
rotated images P_ 6 o/, P_ 12 o/, P_ 18 o/, P_ 2 4 o/ with ro¬ 
tation angles within [0,30°) and interval of 6°. For each 
patch \Fp(/) centered at pixel p , we extract its rotated ver¬ 
sions ty p (ReI) (0 G {—6°, —12°, —18°, —24°}) from the 
rotated images. By feeding these patches into CNN, we can 
predict the probabilities of motion kernels for patch 'ftp (I) 
using each patch based on Eqn. 0: 

P(m = (l, o)\'f> p (I)) = P(m = (l, 
P(m=(l,o + 6°)\y p (I)) = P(m= (1,o)\V p (R- 6 oI)), 
P(m = (l,o + 12°)\%(I)) = P(m = (Z,o)|* p (iJ_ 12 ./)), 
P(m = (l, o + 18°)|tf p (J)) = P(m = (l, o)|^ p (P_ 18 oJ)), 
P(m = (Z, o + 24°)|M/p(7)) = P(m = (l, o)\%(R-wI)), 


where 1 G ^,o G S°. By concatenating all the above es¬ 
timations from one patch and its rotated versions, we can 
therefore predict the motion distribution in an extended mo¬ 
tion kernel set of CNN: P(m = (/, o)|\F p (/)), o G S° ext — 
{0°, 6°, 12°, • • • , 174°}, l G S l . After motion kernels ex¬ 
tension, we can totally predict probabilities of 361 candi¬ 
date motion kernel^] for an image patch by CNN, which is 
almost 5 times of the number of candidate motion kernels in 
S predicted by CNN. Note that this process does not require 
the CNN retraining, but just feed this image and its rotated 
versions to our learned CNN. 



(a) Blurry image (b) Estimation without CNN motion (c) Estimation with CNN motion 

kernel set extension (MSE = 10.3) kernel set extension (MSE = 8.4) 


Figure 6. Effect of CNN motion kernel set extension. 

Figure [6] shows an example of motion kernel estimation 
without and with CNN motion kernel set extension. In this 
example, we synthetically generate the motion blur using a 
camera motion. As shown in Fig. |6fb), the estimated mo¬ 
tion kernels suffer from blocky artifacts in the blue rectangle 
because all pixels in it are predicted to have the same orien¬ 
tation due to the large quantization interval of motion orien¬ 
tations. By extending the motion kernel set of CNN, we can 
predict more accurate motion kernels shown in Fig. [6jc). 
The mean squared error (MSE) w.r.t. ground-truth motion 
kernels is reduced from 10.3 to 8.4. 

3. Dense Motion Field Estimation by MRF 

The CNN predicts distribution of motion kernels in an 
image at the patch level. We now discuss how to fuse these 

3 There are totally 390 possible kernels by combining the motion 
lengths in S l (|S^ | = 13) and motion orientations in S° xt (\S° xt \ = 30). 
But the motion kernels (m = (/, o)} oe s° xt when Z = 1 are all the same, 
we only retain one of them. 
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Orientation: 0-180 



Figure 7. Examples of motion kernel probabilities. The left of (b) show four blurry patches cropped from (a). Each color map on the right 
of (b) shows the probabilities of motion kernels in different motion lengths and orientations estimated for each blurry patch by CNN. Note 
that the high probability regions are local in each map. (c) shows our final motion kernel estimation. 











(b) Result without MRF 
(MSEjmotion = 22.93) 



Figure 8. Example of non-uniform motion kernel estimation, (b) Estimation using the unary term of Eqn. i e., choosing the motion 
kernel with highest confidence for each pixel, (c) Estimation using the full model of Eqn. {6} with motion smoothness constraint, (d) 
Ground-truth motion blur. MSE_motion is an accuracy measurement of motion blur defined in Section 5. 


patch-level motion kernel estimations into a dense field of 
motion kernels for the image. 

Given an image /, we sample 30 x 30 x 3 overlapping 
color patches with a spatial interval of 6 pixels over image 
I. Each patch Sk p (I) produces motion kernel probabilities: 
P(m = (l,0)\* p (I)) (l e S l ,o € S° ext ) by applying the 
CNN. Figure [TJb) shows examples of motion distribution 
maps for four blurry patches, and each of them is sparse and 
composed of one local high probability region. We assume 
that the pixels in patch Sk p (I) share the same motion dis¬ 
tribution. Then each pixel has multiple estimates for each 
motion kernel probability from all patches containing it. For 
a pixel p , we perform weighted average over the multiple 
estimates of each motion kernel probability, and define the 
confidence of motion kernel m = (/, 6) at pixel p as 

C( m p = (l,o)) = 

7F Y1 G <t(\\ X P ~ x q\\ 2 ) p ( m = (5) 

q-.pe^q 

for al ll e S l ,o e S° ext . x p is the coordinate of pixel p. 
As a Gaussian function, G a (\\x p — x q \\ 2 ) imposes higher 
weights on the patch in the summation if its center pixel 
q is closer to pixel p. a is set to 10 in our implementation. 
This means that we trust more the motion prediction from 
the patch containing pixel p closer to its patch center. Z = 
^2 q . pe v G a (| \x p — x q 11) is a normalization constant. 

We further assume that the motion kernels are spatially 
smooth. This is reasonable because the moving objects or 


camera are moving smoothly during capturing image, and 
nearby pixels should have similar motions. Then we esti¬ 
mate the dense motion field M = {m p = (l p ,o p )} pe n 
over image / by optimizing the following MRF model: 

min M ^2[~C{m p = ( l p ,o p )) + 

pen 

^ ^ A [(tip U q ) + ( V p V q ) ] 3 (6) 

qeN(p ) 

where l p e S l ,o p e S° xt , ( u p , v p ) and (u q , v q ) are motion 
vectors m p and m q in Cartesian coordinates that are related 
to (lp,o p ) and (l q ,o q ) by Eqn. 0- N(p) is the neighbor¬ 
hood of p. By minimizing the energy function, the first term 
encourages to choose the motion kernel for each pixel with 
higher confidence estimated by CNN, and the second term 
enforces the smoothness of nearby motion kernels. 

For each pixel, there are 361 motion kernel candidates, 
it is inefficient to optimize the MRF problem with such a 
large number of candidate labels. We therefore generate 
candidate motion kernels for each pixel by selecting the top 
20 motion kernels with highest confidence values, together 
with 30 sampled motion candidates from the remaining can¬ 
didates to make the motion kernel candidate set for each 
pixel both prominent and diverse. Since the candidate label 
sets are spatially varying, we cannot use the off-the-shelf 
graph cut toolbox (24), we therefore optimize the energy 
by max-product belief propagation algorithm. Predicting 
dense motion blur for an image of size 300 x 400 takes 
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around 80 seconds using CPU including computing patch- 
level motion distributions by CNN. 

Figure [8] shows an example of motion blur estimation. 

As shown in Fig.[8jb, c), the full MRF model can effectively 
remove the noisy estimates in Fig. [8jb) using smoothness 
term, and quantitative results are significantly improved. 

4. Non-Uniform Motion Deblurring 

With the dense non-uniform motion kernels estimated 
by CNN, we now deconvolve the blurry image to estimate 
the sharp image. It is challenging to deconvolve the image 
blurred by non-uniform motion blur. We adapt the uniform 
deconvolution approach in 132] to the non-uniform decon¬ 
volution problem. The non-uniform deconvolution is mod¬ 
eled as optimizing: 

min / ^||fc(M) *I-0\\\ - (7) 

ievt 

where O is the observed blurry image, Ri is an operator to 
extract the patch located at i from an image. P(-) is the 
prior distribution of natural image patches, which is mod¬ 
eled as a Gaussian mixture model learned from natural im¬ 
age patches 132]. 

Different to uniform deblur in (32), the first term in 
Eqn.Q is modeled for non-uniform motion blur. We op¬ 
timize the above problem by half-quadratic splitting algo¬ 
rithm, i.e., optimizing: min/ ^ z .y^\\k(M) * I — 0||| + 
\\RJ ~ Zi\\l - log(P(*<))), where auxiliary vari¬ 
ables {zi} are introduced. We iteratively optimize I and 
{zi} by increasing /3. In the iterations, we need to opti¬ 
mize the following two sub-problems. (1) By fixing {zi}, 
we optimize sharp image: min/|||&(M) * I — 0111 + 
Z)ien(f \\RiI ~ Zi\\l). (2) By fixing I, we optimize {z t }: 
min Zi f || RJ - Zi\%- log (P(zi)),i € 12. 

For sub-problem (1), the blur kernels k(M) are non- 
uniform and determined by spatially varying motion vectors 
M. By re-writing the non-uniform convolution as matrix- 
vector multiplication (i.e.,fc(M)*7 = Km I ), we optimize 
sharp image by solving the linear equations deduced by set¬ 
ting the gradients of cost in sub-problem (1) to zeros: 

[a k t m k m +/? Y^ RT i R i)V = xk m°+p(52 R ? Zi )■ (8) 

We solve these linear equations using a conjugate gradient 
algorithm. In the implementation, all the involved matrix- 
vector multiplications can be efficiently implemented by 
convolutions or local operations around each pixel. Rf z 
is an operation to put the patch 2 back to the region where 
it was extracted. The sub-problem (2) can be optimized fol¬ 
lowing (32l . In implementation, we set the patch size to 
8 x 8, A = 2 x 10 5 , and /? is increased from 50 to 3200 with 
a ratio of 2 in 7 iterations of alternative optimizations. 


Table 1. Comparison of motion kernel estimation on 15 test images 
with synthetic motion blur. “BlurSpect” is based on the approach 
in (T|. “SLayerRegr” is the extension of approach in (3]|. 


Methods 

DL MRF 

DL noMRF 

DL noLE 

BlurSpect 

SLayerRegr 

MSE_motion 

7.83 

16.35 

9.01 

44.56 

65.10 

PSNR_motion 

44.55 

37.14 

43.17 

26.58 

22.70 


5. Experiments 

To evaluate the quantitative accuracy of our approach 
for non-uniform motion kernel estimation, we gener¬ 
ate 15 synthetic blurred images with ground-truth non- 
uniform motion kernels caused by camera motions (rota¬ 
tion and translation). The examples shown in Figs. [5} 
[8] are from this synthetic image set. Given the es¬ 
timated motion blur kernels M = {u p ,v p } pe n and 
ground-truth motion blur kernels M gt = {u gt ,v gt } pe n 
in the Cartesian coordinate system, we measure the ac¬ 
curacy of the estimated motion kernel by the mean- 
squared-error (MSE_motion): MSE_motion(M, M gt ) = 
2 p| EpentK - u p ) 2 + ( v p ~ v p) 2 \ and P eak signal-to- 
noise ratio (PSNR_motion): PSNR_motion(M, M gt ) = 
— 10 log MSE - m Q t 2 lon ( M ’ M9 ) ? djnax = 25 is the maximum 

U'max 

motion length. 

Figure [9] presents four examples with strongly non- 
uniform motion blur captured for scenes with complex 
depth layers. The first three examples are real-captured 
blurry images, and the final example is a synthetic blurry 
image. All these examples show that our CNN-based ap¬ 
proach can effectively predict the spatially varying motion 
kernels. 

In Table 1, we evaluate and compare our approach to the 
other approaches for non-uniform motion kernel estimation. 
“DL_noMRF” is our approach using only the unary term in 
Eqn. “DL_noLE” is our MRF-based approach with¬ 
out using the motion kernel set extension. “DLJVtRF” is 
our full estimation approach. “BlurSpect” is the approach 
proposed in m. It was originally designed for estimating 
horizontal or vertical motion blur, and we extend it to esti¬ 
mate motion kernels with orientations in S° xt by the tech¬ 
nique in Section [Z2] “SLayerRegr” is an extended version 
of approach in [3|. The original approach learns a logis¬ 
tic regressor to estimate discrete motion kernels in horizon¬ 
tal direction using hand-crafted features. To predict mo¬ 
tion kernels in other directions, we implement 13 ] using the 
same features and learn SVMs for predicting 73 motion ker¬ 
nels in S, then extend motion kernel set by the method 
in Section |2.2| “SLayerRegr” can be seen as a learning 
machine with a single layer of hand-crafted features. As 
shown in Table[I} both “BlurSpect” and “SLayerRegr” per¬ 
form poorly on estimating the challenging non-uniform mo¬ 
tion blur with diverse motion lengths and orientations. Our 
approach can effectively estimate the motion kernels with 
average MSE_motion 7.83 and PSNR_motion 44.55. More- 
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Figure 9. Examples on motion kernel estimation. The first three columns are real blurry images, the last column shows a synthetic picture 
with camera rotation (MSE_motion = 9.9). 



Figure 10. Examples of non-uniform motion deblurring. The first and second columns show the blurry images and our results. The third 
and fourth columns show the results of methods in lfl8l l26l l27l [29l using their source codes. These examples are challenging because the 
motion blur kernels are strongly non-uniform and the scenes are complex. Our estimated motion blur fields are shown in Figs.|9|[l2|. 


over, the motion kernel set extension and motion smooth¬ 
ness constraint significantly improve the accuracy of motion 
kernel estimation. 


deblurring approaches l|T8l 27]], for which the source codes 
are available. Except for ours, none of these methods han¬ 
dles the non-uniform blur in a satisfying manner for these 
examples. Our approach estimates more accurate motion 
blur kernels, which enables us to produce better final de- 


Figure [TO] compares deblurring results of our approach, 
non-uniform deblurring approaches |26 t [29 ] and uniform 
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Table 2. Accuracies of motion kernel estimation and blur removal on 15 test images with synthetic motion blur. “BlurSpect” is based on 
the approach in |T). “SLayerRegr” is the extension of approach in (5). “MSE_ker” is an error for non-uniform blur kernel estimation using 
the average MSE of blur kernels across image pixels. “PSNR.deblur” is the PSNR of the final deblurred results. The number in each table 
cell is the mean value over the image set. 



DL MRF 

DL noMRF 

DL noLE 

BlurSpect 

SLayerRegr 

TwoPhase 1271 

MargLike [18] 

NonUnif (26] 

UnNatural (29] 

MSE_ker 

0.024 

0.029 

0.041 

0.250 

0.127 

0.108 

0.119 

0.193 

0.165 

PSNR_deblur 

24.81 

24.66 

24.61 

21.72 

19.04 

21.26 

18.49 

20.65 

21.33 



Input image Our result Result of [13] Our estimated motion blur Motion blur by [13] 

Figure 11. Comparison to CD. Our CNN can better predict the different motion layers. The deblurring result of LED is over-sharpened 
and image details are removed, while our result is visually more natural. 



m '- '■ I 

(f) UnNatural [29] 
(MSE_ker = 0.165) 


(a) Blurred image with 
motion kernels 


(b) Our result 
(MSE_ker = 0.025) 


(c) BlurSpect 
(MSE_ker = 0.25) 


Figure 12. Comparison of motion kernel estimation. “BlurSpect” and “SLayerRegr” are based on the methods in [ TJ and [31 respectively. 


blurring results. The method in 111! is an effective approach 
for motion deblurring. Because its source code is not avail¬ 
able, we directly compare it on examples of ED in Fig. [TT] 
Our approach can better “recognize” the complex motions. 
The deblurring result of LTD is commonly over-sharpened, 
but our deblurring result is visually more natural. 

In Table [2] we qualitatively compare our method to the 
state-of-the-art non-blind debluring approaches for both the 
motion blur kernel estimation and the final deblurred re¬ 
sults. We define an error of “MSE_ker” for non-uniform 
motion blur estimation using average MSE of blur kernels 
across pixels in an image, and the MSE of each pixel is de¬ 
fined by the mean per-element squared difference between 
the estimated and ground-truth kernels after aligning ker¬ 
nels by centers. Contrary to the “MSE_motion” that mea¬ 
sures the kernel error in the linear motion space, this error 
term directly measures the kernel differences in the spatial 
domain. We also evaluate the deblurring result by the PSNR 
of the deblurred image (denoted as “PSNR_deblur”) w.r.t. 
the ground-truth clean image. All the values in Tableware 
the mean values over the image set. We can not qualita¬ 


tively compare to the approach in (L3l because the source 
codes are not available. These results clearly show that our 
approach can produce significantly better results both in the 
motion blur kernel estimation and the motion blur removal 
than the compared state-of-the-art approaches. 

Figure [12] shows an example of motion blur estimation 
by different non-uniform blur estimation approaches. Our 
approach can produce significantly better non-uniform mo¬ 
tion blur field than the compared approaches. 

6. Conclusion 

In this paper, we have proposed a novel CNN-based non- 
uniform motion deblurring approach. We learn an effective 
CNN for estimating motion kernels from local patches. Us¬ 
ing an MRF model, we are able to well predict the non- 
uniform motion blur field. This leads to state-of-the-art mo¬ 
tion deblurring results. In the future, we are interested in 
designing a CNN for estimating the general non-uniform 
blur kernels. We are also interested in designing an CNN 
system that can estimate and remove general non-uniform 
blurs in a single framework. 
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